Systems and methods for storage modeling and costing

ABSTRACT

The present invention provides systems and methods for data storage. A hierarchical storage management architecture is presented to facilitate data management. The disclosed system provides methods for evaluating the state of stored data relative to enterprise needs by using weighted parameters that may be user defined. Also disclosed are systems and methods evaluating costing and risk management associated with stored data.

CROSS-REFERENCE TO RELATED APPLICATIONS

Any and all priority claims identified in the Application Data Sheet, orany correction thereto, are hereby incorporated by reference under 37CFR 1.57.

This application is related to the following patents and pendingapplications, each of which is hereby incorporated herein by referencein its entirety:

-   -   application Ser. No. 09/354,058, titled HIERARCHICAL BACKUP AND        RETRIEVAL SYSTEM, filed Jul. 15, 1999, now U.S. Pat. No.        7,395,282;    -   U.S. Pat. No. 6,418,478, titled PIPELINED HIGH SPEED DATA        TRANSFER MECHANISM, issued Jul. 9, 2002;    -   application Ser. No. 09/610,738, titled MODULAR BACKUP AND        RETRIEVAL SYSTEM USED IN CONJUNCTION WITH A STORAGE AREA        NETWORK, filed Jul. 6, 2000, now U.S. Pat. No. 7,035,880;    -   U.S. Pat. No. 6,542,972, titled LOGICAL VIEW AND ACCESS TO        PHYSICAL STORAGE IN MODULAR DATA AND STORAGE MANAGEMENT SYSTEM,        issued Apr. 1, 2003;    -   U.S. Pat. No. 6,658,436, titled LOGICAL VIEW AND ACCESS TO DATA        MANAGE BY A MODULAR DATA AND STORAGE MANAGEMENT SYSTEM, issued        Dec. 2, 2003;    -   application Ser. No. 10/658,095, titled DYNAMIC STORAGE DEVICE        POOLING IN A COMPUTER SYSTEM, filed Sep. 9, 2003, now U.S. Pat.        No. 7,130,970;    -   application Ser. No. 10/262,556, titled METHOD FOR MANAGING        SNAPSHOTS GENERATED BY AN OPERATING SYSTEM OR OTHER APPLICATION,        filed Sep. 30, 2002, now U.S. Pat. No. 7,346,623;    -   application Ser. No. 60/460,234, SYSTEM AND METHOD FOR        PERFORMING STORAGE OPERATIONS IN A COMPUTER NETWORK, filed Apr.        3, 2003; and    -   application Ser. No. 10/877,831, HIERARCHICAL SYSTEM AND METHOD        FOR PERFORMING STORAGE OPERATIONS IN A COMPUTER NETWORK, filed        Jun. 25, 2004, now U.S. Pat. No. 7,454,569.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialthat is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosures, as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightrights whatsoever.

BACKGROUND Field of the Invention

The invention disclosed herein relates generally to performing storageoperations on electronic data in a computer network. More particularly,the present invention relates to integrating storage-related operationsfor a computer network according to a specified arrangement of storageoperation cells.

Description of the Related Art

Current storage management systems employ a number of different methodsto perform storage operations on electronic data. For example, data canbe stored in primary storage as a primary copy or in secondary storageas various types of secondary copies including, as a backup copy, asnapshot copy, a hierarchical storage management copy (“HSM”), as anarchive copy, and as other types of copies.

A primary copy of data is generally a production copy or other “live”version of the data which is used by a software application and isgenerally in the native format of that application. Primary copy datamay be maintained in a local memory or other high-speed storage devicethat allows for relatively fast data access if necessary. Such primarycopy data is typically intended for short term retention (e.g., severalhours or days) before some or all of the data is stored as one or moresecondary copies, for example to prevent loss of data in the event aproblem occurred with the data stored in primary storage.

Secondary copies include point-in-time data and are typically forintended for long-term retention (e.g., weeks, months or years dependingon retention criteria, for example as specified in a storage policy asfurther described herein) before some or all of the data is moved toother storage or discarded. Secondary copies may be indexed so users canbrowse and restore the data at another point in time. After certainprimary copy data is backed up, a pointer or other location indicia suchas a stub may be placed in the primary copy to indicate the currentlocation of that data.

One type of secondary copy is a backup copy. A backup copy is generallya point-in-time copy of the primary copy data stored in a backup formatas opposed to in native application format. For example, a backup copymay be stored in a backup format that is optimized for compression andefficient long-term storage. Backup copies generally have relativelylong retention periods and may be stored on media with slower retrievaltimes than other types of secondary copies and media. In some cases,backup copies may be stored at on offsite location.

Another form of secondary copy is a snapshot copy. From an end-userviewpoint, a snapshot may be thought as an instant image of the primarycopy data at a given point in time. A snapshot generally captures thedirectory structure of a primary copy volume at a particular moment intime, and also preserves file attributes and contents. In someembodiments, a snapshot may exist as a virtual file system, parallel tothe actual file system. Users typically gain a read-only access to therecord of files and directories of the snapshot. By electing to restoreprimary copy data from a snapshot taken at a given point in time, usersmay also return the current file system to the prior state of the filesystem that existed when the snapshot was taken.

A snapshot may be created instantly, using a minimum of file space, butmay still function as a conventional file system backup. A snapshot maynot actually create another physical copy of all the data, but maysimply create pointers that are able to map files and directories tospecific disk blocks.

In some embodiments, once a snapshot has been taken, subsequent changesto the file system typically do not overwrite the blocks in use at thetime of snapshot. Therefore, the initial snapshot may use only a smallamount of disk space needed to record a mapping or other data structurerepresenting or otherwise tracking the blocks that correspond to thecurrent state of the file system. Additional disk space is usually onlyrequired when files and directories are actually modified later.Furthermore, when files are modified, typically only the pointers whichmap to blocks are copied, not the blocks themselves. In someembodiments, for example in the case of copy-on-write snapshots, when ablock changes in primary storage, the block is copied to secondarystorage before the block is overwritten in primary storage and thesnapshot mapping of file system data is updated to reflect the changedblock(s) at that particular point in time. An HSM copy is generally acopy of the primary copy data, but typically includes only a subset ofthe primary copy data that meets a certain criteria and is usuallystored in a format other than the native application format. Forexample, an HSM copy might include only that data from the primary copythat is larger than a given size threshold or older than a given agethreshold and that is stored in a backup format. Often, HSM data isremoved from the primary copy, and a stub is stored in the primary copyto indicate its new location. When a user requests access to the HSMdata that has been removed or migrated, systems use the stub to locatethe data and often make recovery of the data appear transparent eventhough the HSM data may be stored at a location different from theremaining primary copy data.

An archive copy is generally similar to an HSM copy, however, the datasatisfying criteria for removal from the primary copy is generallycompletely removed with no stub left in the primary copy to indicate thenew location (i.e., where it has been moved to). Archive copies of dataare generally stored in a backup format or other nonnative applicationformat. In addition, archive copies are generally retained for very longperiods of time (e.g., years) and in some cases are never deleted. Sucharchive copies may be made and kept for extended periods in order tomeet compliance regulations or for other permanent storage applications.

In some embodiments, application data over its lifetime moves from moreexpensive quick access storage to less expensive slower access storage.This process of moving data through these various tiers of storage issometimes referred to as information lifecycle management (“ILM”). Thisis the process by which data is “aged” from more forms of secondarystorage with faster access/restore times down through less expensivesecondary storage with slower access/restore times, for example, as thedata becomes less important or mission critical over time.

Examples of various types of data and copies of data are furtherdescribed in the above-referenced related applications that are herebyincorporated by reference in their entirety. One example of a systemthat performs storage operations on electronic data that produce suchcopies is the QiNetix storage management system by CommVault Systems ofOceanport, N.J.

The QiNetix system leverages a modular storage management architecturethat may include, among other things, storage manager components, clientor data agent components, and media agent components as furtherdescribed in U.S. patent application Ser. No. 10/818,749 which is herebyincorporated herein by reference in its entirety. The QiNetix systemalso may be hierarchically configured into backup cells to store andretrieve backup copies of electronic data as further described in U.S.patent application Ser. No. 09/354,058 which is hereby incorporated byreference in its entirety.

Backup systems may be improved by providing systems and methods tohierarchically configure backup cells to perform certain other types ofstorage operations including snapshot copies, HSM copies, archivecopies, and certain other types of copies of electronic data.

A further shortcoming of prior art storage systems relates to the lackof meaningful integration between and among HSM systems and StorageResource Management (“SRM”) systems. SRM systems are typically concernedwith monitoring the health, status, and other information associatedwith primary copies of data (e.g., live or production line copies). Forexample, an SRM application may monitor the capacity of a volume storinga primary copy of data and the rate of data transfer relative to theavailable storage of that volume. In contrast, HSM systems are typicallyconcerned with data transport to secondary storage, such as theparameters and particulars associated with the storage and migration ofsecondary copy data as opposed to monitoring of information associatedwith primary copy data. HSM systems are thus generally directed tosecondary copies and other archival data volumes. For example, an HSMsystem may be directed to migrating or otherwise transferring data to asecondary copy.

Existing SRM systems, however, do not meaningfully interoperate withexisting HSM systems across an enterprise or network. For example,although an SRM system may monitor the available space of a primarystorage volume and note when its capacity is being reached (orexceeded), the SRM system itself is typically incapable of initiatingany substantive corrective or preventative action. Similarly, because anHSM system is typically concerned only with migrating and managingsecondary copy data to secondary volumes and data stores, it istherefore generally unaware of conditions or problems associated withprimary copy data. In the past, to address this problem, users haveimplemented two distinct SRM and HSM systems and manually handled issuesdiscovered by SRM systems when HSM support or other input is required orvice versa. Thus, in view of the foregoing, it would be desirable toprovide systems and methods that more meaningfully integrate SRM and HSMcomponents in storage management systems to perform autonomous orpseudo-autonomous storage operations and other operations.

SUMMARY

The present invention addresses, among other things; the shortcomingsdiscussed above in performing storage operations on electronic data in acomputer network.

In accordance with some aspects of the present invention, a computerizedsystem is provided for performing storage operations on electronic data,the system including: a first storage operation cell including a firststorage manager component, a media agent component, and a data agentcomponent; and a second storage manager component programmed to controlthe first storage manager component.

In some embodiments disclosed herein, data from a plurality of storageoperation cells is received and normalized to identify storagecharacteristics, attributes, or methods. In some embodiments, data froman SRM storage operation cell is correlated to data from an HSM storageoperation cell, for example, for identifying identical network elementshaving a different name in each storage operation cell.

In some embodiments, the systems and methods disclosed herein detect andmitigate risks in a storage network. The system may identify astorage-related characteristic and may perform an action if thecharacteristic satisfies a particular risk criteria. Risk criteria maybe indicated by storage policies, storage definitions, or otherinformation located in a storage manager database or other component.

In some embodiments disclosed herein, the system generates an indicationof a state of storage in a plurality of storage operation cells. Theindication may include a metric associated with a state of storage, suchas a level of data protection or data availability. In some embodiments,the metric may be generated according to values or weights associatedwith a service level agreement (“SLA”).

In some embodiments, the system may calculate data costing informationand other information including information associated with the cost ofstoring data and data availability associated with storage operationcells. The system may identify network elements, associatecharacteristics or metrics with the network elements, receive additionaldata, such as SRM or HSM data, from storage operation cells, andcorrelate the additional data with the network elements to calculate acost of data or an availability of data. In some embodiments, data maybe identified according to user, department, project, or otheridentifier. In other embodiments, data availability or data cost iscompared to an SLA. In some embodiments, a prediction of media usage isgenerated according to data use, availability, or cost.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is illustrated in the figures of the accompanying drawingswhich are meant to be exemplary and not limiting, in which likereferences are intended to refer to like or corresponding parts, and inwhich:

FIG. 1 is a block diagram of a storage operation cell in a system toperform storage operations on electronic data in a computer networkaccording to an embodiment of the invention.

FIG. 2 is a block diagram of a hierarchically organized group of storageoperation cells in a system to perform storage operations on electronicdata in a computer network according to an embodiment of the invention.

FIG. 2A is an illustrative interface screen showing a unified view ofHSM and SRM data constructed in accordance with one embodiment of thepresent invention.

FIG. 2B is a portion of the illustrative screen of FIG. 2A, constructedin accordance with one embodiment of the present invention.

FIG. 2C depicts and illustrative trending report constructed inaccordance with one embodiment of the present invention.

FIG. 2D is a portion of the trending report shown in FIG. 2C inaccordance with one embodiment of the present invention.

FIG. 2E depicts a graphical user interface suitable for creatingtrending and other reports in accordance with one embodiment of thepresent invention.

FIG. 3 is a block diagram of a hierarchically organized group of storageoperation cells in a system to perform SRM and HSM storage operations onelectronic data in a computer network according to an embodiment of theinvention.

FIG. 4 is a flow diagram of a method to integrate data from storageoperation cells according to an embodiment of the invention.

FIG. 5 is a flow diagram of a method to calculate storage metricsassociated with storage operation cells according to an embodiment ofthe invention.

FIG. 6 illustrates an exemplary graphical representation illustratinglevel of certain performance metric in accordance with an embodiment ofthe invention.

FIG. 7 is an illustrative graphical user interface that may be employedto select certain storage parameters.

FIG. 8 is a flow diagram of a method to calculate data costing and dataavailability associated with storage operation cells according to anembodiment of the invention.

FIG. 9 depicts a graphical user interface constructed in accordance withone embodiment of the present invention for creating cost categories.

FIG. 10 is a flow diagram illustrating some of the steps involved indetermining the cost of data storage in accordance with one embodimentof the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

With reference to FIGS. 1 through 10, embodiments and features of theinvention are presented. FIG. 1 illustrates a block diagram of a storageoperation cell 50 that may perform storage operations on electronic datain a computer network in accordance with an embodiment of the presentinvention. As shown, storage operation cell 50 may generally include astorage manager 100, a data agent 95, a media agent 105, a storagedevice 115, and, in some embodiments, may include certain othercomponents such as a client 85, a data or information store 90,databases 110 and 111, jobs agent 120, an interface module 125, and amanagement agent 130. Such system and elements thereof are exemplary ofa modular backup system such as the CommVault QiNetix system, and alsothe CommVault GALAXY backup system, available from CommVault Systems,Inc. of Oceanport, N.J., and further described in U.S. patentapplication Ser. No. 09/610,738 which is incorporated herein byreference in its entirety.

A storage operation cell, such as cell 50, may generally includecombinations of hardware and software components associated withperforming storage operations on electronic data. Exemplary storageoperation cells according to embodiments of the invention may include,as further described herein, CommCells as embodied in the QNet storagemanagement system and the QiNetix storage management system by CommVaultSystems of Oceanport, N.J. According to some embodiments of theinvention, storage operations cell 50 may be related to backup cells andprovide some or all of the functionality of backup cells as described inapplication Ser. No. 09/354,058. However, in certain embodiments,storage operation cells may also perform additional types of storageoperations and other types of storage management functions that are notgenerally offered by backup cells.

In accordance with certain embodiments of the present invention,additional storage operations performed by storage operation cells mayinclude creating, storing, retrieving, and migrating primary data copiesand secondary data copies (which may include, for example, snapshotcopies, backup copies, HSM copies, archive copies, and other types ofcopies of electronic data). In some embodiments, storage operation cellsmay also provide one or more integrated management consoles for users orsystem processes to interface with in order to perform certain storageoperations on electronic data as further described herein. Suchintegrated management consoles may be displayed at a central controlfacility or several similar consoles distributed throughout multiplenetwork locations to provide global or geographically specific networkdata storage information.

In some embodiments, storage operations may be performed according to astorage policy. A storage policy is generally a data structure or otherinformation source that includes a set of preferences and other storagecriteria for performing a storage operation. The preferences and storagecriteria may include, but are not limited to, a storage location,relationships between system components, network pathway to utilize,retention policies, data characteristics, compression or encryptionrequirements, preferred system components to utilize in a storageoperation, and other criteria relating to a storage operation. Thus, astorage policy may indicate that certain data is to be stored in aspecific storage device, retained for a specified period of time beforebeing aged to another tier of secondary storage, copied to secondarystorage using a specified number of streams, etc. A storage policy maybe stored to a storage manager database 111, to archive media asmetadata for use in restore operations or other storage operations, orto other locations or components of the system.

A schedule policy may specify when to perform storage operations and howoften and may also specify performing certain storage operations onsub-clients of data and how to treat those sub-clients. A sub-client mayrepresent static or dynamic associations of portions of data of a volumeand are mutually exclusive. Thus, a portion of data may be given a labeland the association is stored as a static entity in an index, databaseor other storage location used by the system. Sub-clients may also beused as an effective administrative scheme of organizing data accordingto data type, department within the enterprise, storage preferences,etc. For example, an administrator may find it preferable to separatee-mail data from financial data using two different sub-clients havingdifferent storage preferences, retention criteria, etc. Storageoperation cells may contain not only physical devices, but also mayrepresent logical concepts, organizations, and hierarchies. For example,a first storage operation cell 50 may be configured to perform HSMoperations, such as data backup or other types of data migration, andmay include a variety of physical components including a storage manager100 (or management agent 130), a media agent 105, a client component 85,and other components as described herein. A second storage operationcell may contain the same or similar physical components, however, itmay be configured to perform SRM operations, such as monitoring aprimary data copy or performing other known SRM operations.

Thus, as can be seen from the above, although the first and secondstorage operation cells are logically distinct entities configured toperform different management functions (i.e., HSM and SRM respectively),each cell may contain the same or similar physical devices in bothstorage operation cells. Alternatively, in other embodiments, differentstorage operation cells may contain some of the same physical devicesand not others. For example, a storage operation cell 50 configured toperform SRM tasks may contain a media agent 105, client 85, or othernetwork device connected to a primary storage volume, while a storageoperation cell 50 configured to perform HSM tasks may instead include amedia agent 105, client 85, or other network device connected to asecondary storage volume and not contain the elements or componentsassociated with and including the primary storage volume. These twocells, however, may each include a different storage manager 100 thatcoordinates storage operations via the same media agents 105 and storagedevices 115. This “overlapping” configuration allows storage resourcesto be accessed by more than one storage manager 100 such that multiplepaths exist to each storage device 115 facilitating failover, loadbalancing and promoting robust data access via alternative routes.

Alternatively, in some embodiments, the same storage manager 100 maycontrol two or more cells 50 (whether or not each storage cell 50 hasits own dedicated storage manager 100). Moreover, in certainembodiments, the extent or type of overlap may be user-defined (througha control console (not shown>> or may be automatically configured tooptimize data storage and/or retrieval.

Data agent 95 may be a software module or part of a software module thatis generally responsible for archiving, migrating, and recovering datafrom client computer 85 stored in an information store 90 or othermemory location. Each client computer 85 may have at least one dataagent 95 and the system can support multiple client computers 85. Insome embodiments, data agents 95 may be distributed between client 85and storage manager 100 (and any other intermediate components (notshown>> or may be deployed from a remote location or its functionsapproximated by a remote process that performs some or all of thefunctions of data agent 95.

Embodiments of the present invention may employ multiple data agents 95each of which may backup, migrate, and recover data associated with adifferent application. For example, different individual data agents 95may be designed to handle Microsoft Exchange data, Lotus Notes data,Microsoft Windows 2000 file system data, Microsoft Active DirectoryObjects data, and other types of data known in the art. Otherembodiments may employ one or more generic data agents 95 that canhandle and process multiple data types rather than using the specializeddata agents described above.

If a client computer 85 has two or more types of data, one data agent 95may be required for each data type to archive, migrate, and restore theclient computer 85 data. For example, to backup, migrate, and restoreall of the data on a Microsoft Exchange 2000 server, the client computer85 may use one Microsoft Exchange 2000 Mailbox data agent 95 to backupthe Exchange 2000 mailboxes, one Microsoft Exchange 2000 Database dataagent 95 to backup the Exchange 2000 databases, one Microsoft Exchange2000 Public Folder data agent 95 to backup the Exchange 2000 PublicFolders, and one Microsoft Windows 2000 File System data agent 95 tobackup the client computer's 85 file system. These data agents 95 wouldbe treated as four separate data agents 95 by the system even thoughthey reside on the same client computer 85.

Alternatively, other embodiments may use one or more generic data agents95, each of which may be capable of handling two or more data types. Forexample, one generic data agent 95 may be used to back up, migrate andrestore Microsoft Exchange 2000 Mailbox data and Microsoft Exchange 2000Database data while another generic data agent may handle MicrosoftExchange 2000 Public Folder data and Microsoft Windows 2000 File Systemdata, etc.

Generally speaking, storage manager 100 may be a software module orother application that coordinates and controls storage operationsperformed by storage operation cell 50. Storage manager 100 maycommunicate with some or all elements of storage operation cell 50including client computers 85, data agents 95, media agents 105, andstorage devices 115, to initiate and manage system backups, migrations,and data recovery.

Storage manager 100 may include a jobs agent 120 that monitors thestatus of some or all storage operations previously performed, currentlybeing performed, or scheduled to be performed by storage operation cell50. Jobs agent 120 may be communicatively coupled with an interfaceagent 125 (typically a software module or application). Interface agent125 may include information processing and display software, such as agraphical user interface (“GUI”), an application program interface(“API”), or other interactive interface through which users and systemprocesses can retrieve information about the status of storageoperations. Through interface 125, users may optionally issueinstructions to various storage operation cells 50 regarding performanceof the storage operations as described and contemplated by the presentinvention. For example, a user may modify a schedule concerning thenumber of pending snapshot copies or other types of copies scheduled asneeded to suit particular needs or requirements. As another example, auser may employ the GUI to view the status of pending storage operationsin some or all of the storage operation cells in a given network or tomonitor the status of certain components in a particular storageoperation cell (e.g., the amount of storage capacity left in aparticular storage device). As a further example, interface 125 maydisplay the cost metrics associated with a particular type of datastorage and may allow a user to determine overall and target costmetrics associated with a particular data type, or certain storageoperation cell 50 or other storage operation as predefined oruser-defined (discussed in more detail below).

Storage manager 100 may also include a management agent 130 that istypically implemented as a software module or application program. Ingeneral, management agent 130 provides an interface that allows variousmanagement components 100 in other storage operation cells 50 tocommunicate with one another. For example, assume a certain networkconfiguration includes multiple cells 50 adjacent to one another orotherwise logically related in a WAN or LAN configuration (not shown).With this arrangement, each cell 50 may be connected to the otherthrough each respective interface module 125. This allows each cell 50to send and receive certain pertinent information from other cells 50including status information, routing information, information regardingcapacity and utilization, etc. These communication paths may also beused to convey information and instructions regarding storageoperations.

For example, a management agent 130 in first storage operation cell 50may communicate with a management agent 130 in a second storageoperation cell 50 regarding the status of storage operations in thesecond storage operation cell. Another illustrative example includes thecase where a management agent 130 in first storage operation cell 50communicates with a management agent 130 in a second storage operationcell to control the storage manager 100 (and other components) of thesecond storage operation cell via the management agent 130 contained inthe storage manager 100.

Another illustrative example is the case where management agent 130 inthe first storage operation cell 50 communicates directly with andcontrols the components in the second storage management cell 50 andbypasses the storage manager 100 in the second storage management cell.If desired, storage operation cells 50 can also be organizedhierarchically such that hierarchically superior cells control or passinformation to hierarchically subordinate cells or vice versa.

Storage manager 100 may also maintain an index cache, a database, orother data structure 111. The data stored in database 111 may be used toindicate logical associations between components of the system, userpreferences, management tasks, some SRM or HSM data or other usefuldata. As further described herein, some of this information may bestored in a media agent database or other local data store according tosome embodiments. For example, the storage manager 100 may use data fromdatabase 111 to track logical associations between media agents 105 andstorage devices 115.

Generally speaking, a media agent 105 may be implemented as softwaremodule that conveys data, as directed by a storage manager 100, betweena client computer 85 and one or more storage devices 115 such as a tapelibrary, a magnetic media storage device, an optical media storagedevice, or any other suitable storage device. In one embodiment, mediaagents 105 may be communicatively coupled with and control a storagedevice 115 associated with that particular media agent. A media agent105 may be considered to be associated with a particular storage device115 if that media agent 105 is capable of routing and storing data toparticular storage device 115.

In operation, a media agent 105 associated with a particular storagedevice 115 may instruct the storage device to use a robotic arm or otherretrieval means to load or eject a certain storage media, and tosubsequently archive, migrate, or restore data to or from that media.Media agents 105 may communicate with a storage device 115 via asuitable communications path such as a SCSI or fiber channelcommunications link. In some embodiments, the storage device 115 may becommunicatively coupled to a data agent 105 via a Storage Area Network(“SAN”).

Each media agent 105 may maintain a index cache, a database, or otherdata structure 110 which stores index data generated during backup,migration, and restore and other storage operations as described herein.For example, performing storage operations on Microsoft Exchange datamay generate index data. Such index data provides a media agent 105 orother external device with a fast and efficient mechanism for locatingdata stored or backed up. Thus, in some embodiments, a storage mangerdatabase 111 may store data associating a client 85 with a particularmedia agent 105 or storage device 115, for example, as specified in astorage policy, while media agent database 110 may indicate wherespecifically the client 85 data is stored in storage device 115, whatspecific files were stored, and other information associated withstorage of client 85 data. In some embodiments, such index data may bestored along with the data backed up in a storage device 115, with anadditional copy of the index data written to index cache 110. The datain index cache 110 is thus readily available for use in storageoperations and other activities without having to be first retrievedfrom the storage device 115.

In some embodiments, certain components may reside and execute on thesame computer. For example, in some embodiments, a client computer 85such as a data agent 95, a media agent 105, or a storage manager 100coordinates and directs local archiving, migration, and retrievalapplication functions as further described in U.S. patent applicationSer. No. 09/610,738. This client computer 85 can function independentlyor together with other similar client computers 85.

FIG. 2 presents a generalized block diagram of a hierarchicallyorganized group of storage operation cells in a system to performstorage operations on electronic data in a computer network inaccordance with an embodiment of the present invention. It will beunderstood that although the storage operation cells generally depictedin FIG. 2 have different reference numbers than the cell 50 shown inFIG. 1, these cells may be configured the same as or similar to thestorage cell 50 depicted in FIG. 1.

As shown, the system illustrated in FIG. 2 may include a master storagemanager component 135 and various other storage operations cells. Asshown, the illustrative embodiment in FIG. 2 includes a first storageoperation cell 140, a second storage operation cell 145, a third storageoperation cell 150, a fourth storage operation cell 155, a fifth storageoperation cell 160, and an nth storage operation cell 165. However, itwill be understood this illustration is only exemplary and that fewer ormore storage operation cells may be present or interconnecteddifferently if desired.

Storage operation cells, such as the ones shown in FIG. 2 may becommunicatively coupled and hierarchically organized. For example, amaster storage manager 135 may be associated with, communicate with, anddirect storage operations for a first storage operation cell 140, asecond storage operation cell 145, a third storage operation cell 150, afourth storage operation cell 155, a fifth storage operation cell 160,and an nth storage operation cell 165. In some embodiments, the masterstorage manager 135 may not be part of any particular storage operationcell. In other embodiments (not shown), master storage manager 135 mayitself be part of a certain storage operation cell.

Thus, in operation, master storage manager 135 may communicate with amanagement agent of the storage manager of the first storage⋅ operationcell 140 (or directly with the other components of first cell 140) withrespect to storage operations performed in the first storage operationcell 140. For example, in some embodiments, master storage manager 135may instruct the first storage operation cell 140 with certain commandsregarding a desired storage operation such as how and when to performparticular storage operations including the type of operation and thedata on which to perform the operation.

In other embodiments, master storage manager 135 may track the status ofits associated storage operation cells, such as the status of jobs,system components, system, resources, and—other items, by communicatingwith manager agents (or other components) in the respective storageoperation cells. Moreover, master storage manager 135 may track thestatus of its associated storage operation cells by receiving periodicstatus updates from the manager agents (or other components) in therespective cells regarding jobs, system components, system resources,and other items. For example, master storage manager 135 may use methodsto monitor network resources such as mapping network pathways andtopologies to, among other things, physically monitor storage operationsand suggest, for example, alternate routes for storing data as furtherdescribed herein.

In some embodiments, master storage manager 135 may store statusinformation and other information regarding its associated storageoperation cells and other system information in an index cache, databaseor other data structure accessible to manager 135. A presentationinterface included in certain embodiments of master storage manager 135may access this information and present it to users and system processeswith information regarding the status of storage operations, storageoperation cells, system components, and other information of the system.

As mentioned above, storage operation cells may be organizedhierarchically. With this configuration, storage operation cells mayinherit properties from hierarchically superior storage operation cellsor be controlled by other storage operation cells in the hierarchy(automatically or otherwise). Thus, in the embodiment shown in FIG. 2,storage operation cell 145 may control or is otherwise hierarchicallysuperior to storage operation cells 150, 155, 160 and 165. Similarly,storage operation cell 155 may control storage operation cells 160 and165. Alternatively, in some embodiments, storage operation cells mayinherit or otherwise be associated with storage policies, storagepreferences, storage metrics, or other properties or characteristicsaccording to their relative position in a hierarchy of storage operationcells.

Storage operation cells may also be organized hierarchically accordingto function, geography, architectural considerations, or other factorsuseful or desirable in performing storage operations. For example, inone embodiment, storage operation cell 140 may be directed to createsnapshot copies of primary copy data, storage operation cell 145 may bedirected to create backup copies of primary copy data or other data.Storage operation cell 140 may represent a geographic segment of anenterprise, such as a Chicago office, and storage operation cell 145 mayrepresents a different geographic segment, such as a New York office. Inthis example, the second storage operation cells 145, 150, 155, 160, and165 may represent departments within the New York office. Alternatively,these storage operation cells could be further divided by functionperforming various types of copies for the New York office or loadbalancing storage operations for the New York office.

As another example, and as previously described herein, differentstorage operation cells directed to different functions may also containthe same or a subset of the same set of physical devices. Thus, onestorage operation cell in accordance with the principles of the presentinvention may be configured to perform SRM operations (such as a QSMcell in the CommVault QiNetix system) and may contain the same, similaror a subset of the same physical devices as a cell configured to performHSM or other types of storage operations. Each storage operation cellmay, however, share the same parent or, alternatively, may be located ondifferent branches of a storage operation cell hierarchy tree. Forexample, storage operation cell 140 may be directed to SRM operationswhereas storage operation cell 155 may be directed to HSM operations.Similarly, storage operation cell 150 may be configured to perform SRMoperations and storage operation cell 155 may be configured to performHSM operations. Those skilled in the art will recognize that a widevariety of such combinations and arrangements of storage operation cellsare possible to address a broad range of different aspects of performingstorage operations in a hierarchy of storage operation cells.

In some embodiments, hierarchical organization of storage operationcells facilitates, among other things, system security and otherconsiderations. For example, in some embodiments, only authorized usersmay be allowed to access or control certain storage operation cells. Forexample, a network administrator for an enterprise may have access tomany or all storage operation cells including master storage manager135. But a network administrator for only the New York office, accordingto a previous example, may only have access to storage operation cells145-165, which form the New York office storage management system.

The hierarchical organization of storage operation cells may facilitatestorage management planning and decision-making. For example, in someembodiments, data and other information related to HSM storage operationcells and SRM storage operation cells may be normalized (e.g., convertedinto a common format with normalized identifiers and other information)and combined by a master storage manager component or otherhierarchically superior (or, if desired, hierarchically subordinate)component, and a user of master storage manager 135 may view the statusof some or all jobs in associated storage operation cells as well as thestatus of each component of the storage operation cells present in thesystem. Users may then plan and make decisions based on this globaldata. For example, a user may view high-level information regardingstorage operations for the entire system, such as job status, componentstatus, resource status (such as network pathways, etc.), and otherinformation. This may be done from an SRM perspective to view primarycopy information or from an HSM perspective to view secondary copyinformation. A user may also drill down through menus or use other meansto obtain more detailed information regarding a particular storageoperation cell or group of storage operation cells.

In other embodiments master storage manager 135 may alert a user such asa system administrator when a particular resource is unavailable orcongested. For example, a particular storage device might be full orrequire additional media. For example, a master storage manager may useinformation from an HSM storage operation cell and an SRM storageoperation cell to present indicia or otherwise alert a user or otherwiseidentify aspects of storage associated with the storage managementsystem and hierarchy of storage operation cells.

Alternatively, a storage manager in a particular storage operation cellmay be unavailable due to hardware failure, software problems, or otherreasons. In some embodiments, master storage manager 135 (or anotherstorage manager within the hierarchy of storage operation cells) mayutilize the global data regarding its associated storage operation cellsto suggest solutions to such problems when they occur (or act as awarning prior to occurrence). For example, master storage manager 135may alert the user that a storage device in a particular storageoperation cell is full or otherwise congested, and then suggest, basedon job and data storage information contained in its database, analternate storage device. Other types of corrective actions based ansuch information may include suggesting an alternate data path to aparticular storage device, or dividing data to be stored among variousavailable storage devices as a load balancing measure or to otherwiseoptimize storage or retrieval time. In some embodiments, suchsuggestions or corrective actions may be performed automatically, ifdesired.

In operation, some of the types of information that may be monitored bythe HSM component for an application such as Microsoft Exchange mayinclude:

-   -   File Systems and File Analysis        -   Distribution by file types        -   Distribution by file sizes        -   Distribution by modification time/access time        -   Distribution by owner, etc.        -   Capacity and Asset Reporting        -   By host        -   By disk        -   By partitions        -   Availability Reporting        -   Disks        -   Hosts        -   Applications    -   Message Level        -   Distribution across mailboxes        -   By Message attachment type        -   By Message age        -   Data Migration report        -   Usage Reporting        -   By Storage Group        -   By Store        -   By Mailbox, etc.

In alternate embodiments, HSM and SRM components may be aware of eachother due to a common database or metabase of information which mayinclude normalized data. Therefore, in those embodiments there is noneed for such information to pass through a master storage manager asthese components may be able to communicate directly with one another.For example, storage operation cell 150 may communicate directly withstorage operation cell 165 and vice versa. This may be accomplishedthrough a direct communications link between the two or by passing datathrough intermediate cells, such as cells 145-160 in this case.

FIGS. 2a and 2b present an exemplary management console 208 suitable foruse with the present invention. As shown, management console 208provides a “unified view” such that a user may observe and manage thestatus of both HSM and SRM data operations. Management console 208 linksSRM data concerning front line primary storage with HSM jobs oroperations and provides a user with an indication of how data may bemigrated or is scheduled to be migrated within the storage system. Asshown, controller 208 may include a primary storage section 205 and asecondary storage section 210. Console 208 may be provided eitherlocally or remotely (e.g., over the Internet) to allow users to monitorand control storage operations from virtually any location.

As shown, section 205 provides information concerning primary storage(e.g., SRM) metrics. For example, section 205 includes a bar graph 206illustrating the amount of free and used space in a selected primaryvolume. This information is also presented numerically in metric chart207 which may include the volume name, total space of the volume, spaceused, free space, job scheduling, billable entity, data protectionlevels, job scheduling, cost category, and types of storage operationsperformed or scheduled.

Section 210 of FIG. 2b , which may represent the bottom portion of theunified screen (with section 205 on top) provides information concerningsecondary storage (e.g., HSM). Section 210 may include charts 184 and212 that provide sub-client and related information. For example chart184, may present, instance, and sub-client information along withassociated scheduling, storage policy information and indicate whethercertain data protection operation have occurred, Chart 212 may providethe last data protection type and the billable entity associated theinstance or sub-client.

Moreover, as shown on the right hand side of FIG. 2A, the unifiedconsole may include a directory tree that allows a user to drill down tovarious parts of the system. For example, the user may examine aparticular one of various user groups by selecting one of the users oruser groups specified in field 182. A user may view alerts by selectingfield 189 or billing or other cost categories by selecting from field183. Certain system reports or schedules may be generated by making theappropriate selection in field 190, and information associated withspecific storage cells and other systems specific modules may be viewedby selecting from among the options in field 193 and 194.

One advantage of using such integrated management console is that itallows network administrators to monitor data management activities anddevelopments on a single integrated display without switching betweenmultiple consoles or learning different interfaces. Another benefit isit reduces the expense associated with training administrators onmultiple products and the problems associated with attempting toadministrate systems that are independent of one another. Theinformation on which this console based may be located in a shareddatabase (e.g., a master storage manager index) that stores commonnormalized information (to prevent the confusing duplication of data).However, it will be understood that in certain embodiments, the HSM andSRM screens may be separate or distributed, if desired.

Event viewer 205 may be configured to allow a user monitor some or allSRM and HSM events within the storage system. Event viewer 205 may beconfigured to display operations based in time, type of event or othercriteria (e.g., most recent first, etc.). In some embodiments, a usermay set certain user preferences to display such as the severity levelsand number of events to view. Event viewer 205 may also support searchqueries, allowing a user to view events by time period, severity or JobID. Such information may be displayed in multiple windows focused onspecific clients, events or software modules can be opened and monitoredsimultaneously. Thus, event viewer 205 provides a customizable tool toallow the administrator to effectively and efficiently monitor andmanage the jobs.

In some embodiments, unified console 208 may support a reportingcapability that allows the generation of a variety of reports, which maybe tailored to a particular aspect of data management. Report types mayinclude: scheduling, event management, media management and data aging.Available reports may also include backup history, data aging history,auxiliary copy history, job history, library and drive, media inlibrary, restore history and storage policy. Such reports may bespecified and created at a certain point in time as a network analysis,forecasting, or provisioning tool. Integrated reports may also begenerated that illustrate SRM and HSM information, storage andperformance metrics, risks and storage costing information. Moreover, incertain embodiments, users may create their own reports based onspecific needs.

Furthermore, in some embodiments, master storage manager 135 (or othernetwork storage manager) may contain programming or other logic directedtoward analyzing the storage patterns and resources of its associatedstorage operation cells and may suggests optimal or alternate locationsor methods of performing storage operations. Thus, for example, storagemanager 135 may monitor or otherwise keep track of the amount ofresources available such as storage media in particular cell group ofcells 50. This allows storage manager 135 to note when the level ofavailable storage media (such as magnetic or optical media) falls below(and/or above) a particular level, so additional storage media be addedor removed as need to maintain a desired level of service. Thisinformation may be retained and examined over a time period of interestand used as a forecasting tool (e.g., for predicting future media usage,future media usage growth, trending analysis, system provisioninginformation, etc.). Such information may be presented to the user in theform of a report (discussed in more detail below).

Master storage manager 135 also may analyze traffic patterns and suggestor automatically route data via a particular route to e.g., to certainfacilitate storage and minimize congestion. For example, based on priortraffic information, master storage manager 135 may suggest that backupdata be sent via a different network segment or to a different storageoperation cell or storage device to prevent resource congestion andavoid delays in access time. In some embodiments, users can directspecific queries to master storage manager 135 to obtain predictionsrelating to storage operations or storage operation information. Suchinformation may be presented to the user in the form of a report(discussed in more detail below).

In some embodiments, the predictions described above may be based on atrending analysis that may be used to predict various network operationsor use of network resources such as network traffic levels, storagemedia use, use of bandwidth of communication links, use of media agentcomponents, etc. The trending analysis may employ any suitableforecasting algorithm useful in predicting future network operations orresource use as described above (typically based on past performance).One example of such an algorithm may include calculating a movingaverage of a certain network operation or resource that takes intoconsideration specific information of a time period of interest. Some ofthe steps involved in such a calculation may include: 1) calculating ageneral moving average, 2) calculating a seasonal index, 3) calculatingthe average index for each day, 4) performing a linear interpolation onthe moving average to obtain a linear formula according to the followingequation:

$\begin{matrix}{m - \frac{{{NE}*{\sum{xy}}} - {\sum{x{\sum y}}}}{{{NE}*{\sum x^{2}}} - \left( {\sum x} \right)^{2}}} & (1)\end{matrix}$

and 5) calculating the value forecast by multiplying the moving averageand seasonal average index. However, any other suitable formula ormethod may be used if desired.

In some embodiments, such trending or forecast information may bepresented to the user in the form of a report. For example, FIG. 2cillustrates a trending report for the expected performance of dataprotection operations. As shown, bar graph 201 may be generated thatincludes data representing past data protection operations 202 (fordates April 19-April 25) and data representing the projected dataprotection operations 203 (dates April 26-April 28) and the amount ofdata subject to a protection operation (204). The projections may begenerated based on the techniques described above. Also shown is chartin FIG. 2d which may display specific daily or other time-basedinformation about the data protection operations, such as agent type(field 216) and the amount of data backed up or projected to be backedup (fields 217 and 218 respectively) during particular periods.

Users may create reports based on certain information available withinthe system. The may be accomplished, for example, using a report consolethat may allow the user to specify the parameters or information to beused in creating the report. Alternatively, in some embodiments, reportsmay be predefined and a user may select a report from a list of reportsin a pull down menu or other list (not shown).

FIG. 2e illustrates a console that may be used to generate a report. Theparticular console displayed generates a media prediction report,however, any other suitable report may be generated if desired. Asshown, FIG. 2E includes period selection field 213, chart options field214 and storage cell selection field 219. In operation, a user mayspecify through field 213 the time period of the prediction (i.e., howfar in time the prediction will cover) and the past data on which tobase the prediction. The user may also specify certain chart optionsthrough field 214 such as a cumulative or trend prediction, the charttype, media to include (e.g., consumed or used). Also a user may specifythrough consoles 219 which storage cells or groups of storage cells toinclude in the prediction. Those skilled in the art will recognize thismay be applied to other system resources such as HSM/SRM usage, networktraffic, primary volume growth, SLA levels, failure rates, hardwareusage, etc.

Certain embodiments of the present invention may employ auto-discoverytechniques that may provide a list of available storage devices andlocations which may be made available (automatically or otherwise) tothe administrator for use in establishing a storage policy. By providinga comprehensive list of possible targets for backup copies (e.g., disk,tape and optical), an administrator has an accurate view of theenvironment which facilitates the creation of a corresponding dataprotection scheme utilizing available resources. One advantage of thisis it tends to reduce the possibility of overlooking possible storagedevices or inadvertently misconfiguring the device. By employingauto-discovery techniques, such information may discovered and storedsuch that the policies associated with a discovered device contain anynecessary connection or other configuration information.

FIG. 3 illustrates a block diagram of a hierarchically organized groupof storage operation cells in a system to perform SRM and HSM storageoperations on electronic data in a computer network in accordance withthe principles of the present invention. As shown, FIG. 3 includes afirst storage operation cell 170, a second storage operation cell 175, athird storage operation cell 180, a client 185 in communication with aprimary volume 190 storing data, a storage manager component 195 incommunication with a storage manager database 200, a media agent 206 incommunication with a secondary storage volume 211, and a master storagemanager component 215 in communication with a master storage managerdatabase 220.

The first storage operation cell 170 may be configured to perform aparticular type storage operation, such as SRM storage operations. Forexample, the first storage operation cell 170 may monitor and performSRM-related calculations and operations associated with primary copydata. Thus, the first storage operation cell 170 includes a clientcomponent 185 in communication with a primary volume 190 for storingdata. For example, client 185 may be directed to using Exchange data,SQL data, Oracle data, or other types of production data used inbusiness applications or other applications and stored in primary volume190. Storage manager component 195 may contain SRM modules or otherlogic directed to monitoring or otherwise interacting with attributes,characteristics, metrics, and other information associated with the datastored in primary volume 190. Storage manager 195 may track and storethis and other information in storage manager database 200 which mayinclude index information. For example, in some embodiments, storagemanager component 195 may track or monitor the amount of available spaceand other similar characteristics of data associated with primary volume190. In some embodiments, as further described herein, storage managercomponent 195 may also issue alerts or take other actions when theinformation associated with primary volume 190 satisfies certaincriteria, such as alert criteria. In some embodiments, this may alsoinclude other storage managers, media agents and databases.

Second storage operation cell 175 is directed to another type storageoperation, such as HSM storage operations. For example, the secondstorage operation cell 175 may perform backups, migrations, snapshots,or other types of HSM-related operations known in the art. For example,in some embodiments, data is migrated from faster and more expensivestorage such as magnetic storage (i.e., primary storage) to lessexpensive storage such as tape storage (i.e., secondary storage).

Thus, the second storage operation cell 175 includes client component186 in communication with the primary volume 191 storing data. In someembodiments, client component 186 and primary volume 191 are the samephysical devices as the client component 185 and primary volume 190 inthe first storage operation cell 170. Similarly, in some embodiments,the storage manager component 196 and database 201 (which may includeindex information) in the second storage operation cell 175 are the samephysical devices as the storage manager component and index in the firststorage operation cell 170. Storage manager component 196, however,typically also contains HSM modules or other logic associated with thesecond storage operation cell 175 directed to performing HSM storageoperations on primary volume 191 data.

Storage operation cell 175 therefore may also contain media agent 206and a secondary storage volume 211 configured to performing HSM-relatedoperations on primary copy data. For example, storage manager 196 maymigrate primary copy data from primary volume 191 to secondary volume211 using media agent 206. Storage manager 196 may also track and storesinformation associated with primary copy migration and other similarHSM-related operations in storage manager database 201. For example, insome embodiments, storage manager component 196 may direct HSM storageoperations on primary copy data according to according to a storagepolicy associated with the primary copy 191 and stored in the index 201.In some embodiments, storage manager 196 may also track where primarycopy information is stored, for example in secondary storage 211.

Storage operation cell 180 may include a master storage manager 215 anda master storage manager 220. In some embodiments (not shown),additional storage operation cells may be located hierarchically inbetween storage operation cell 180 and storage operation cells 170 and175. In some embodiments, additional storage operation cellshierarchically superior to operation cell 180 may also be present in thehierarchy of storage operation cells.

In some embodiments, storage operation cells 170 and 175 may beconnected by communications link 197, which may be any suitablecommunications link such as a fiber channel or SCSI connection thatallows cells 170 and 175 to communicate directly with one another (i.e.,without necessarily directly or indirectly involving storage cell 180).This may be accomplished, for example, by storage manager 195communicating with storage manager 196 via link 197. This allows cells170 and 175 to share information such status or operationalcharacteristics to one another such as availability on primary orsecondary storage, trigger events based on such information, etc. Thisallows for the direct transfer of stored data to from and from the cells(via link 197) without the need to communicate with or pass data throughmaster storage manager 215.

Storage operation cell 180 is also generally directed to performing atype of storage operation, such as integration of SRM and HSM data fromother storage operation cells, such as storage operation cells 170 and175. In some embodiments, the third storage operation cell 180 alsoperforms other types of storage operations and might also be directed toHSM, SRM, or other types of storage operations.

Turning to FIG. 4, a flow diagram of a method to integrate data fromstorage operation cells according to one embodiment of the invention ispresented. For example, the method of FIG. 4 can be applied tointegrating SRM data from storage operation cell 170 and HSM data fromstorage operation cell 175.

The system receives data from storage operation cells, step 225. In someembodiments, however, hierarchically equivalent or subordinate cells mayalso receive data from other cells. Generally speaking, however, ahierarchically superior cell, such as storage operation cell 180,receives data from hierarchically subordinate cells, such as cells 170and 175. For example, storage manager 215 may receive SRM data fromstorage manager 195 in storage operation cell 170 and HSM data from thestorage manager 196 in storage operation cell 175. Storage manager 215may receive, from storage operation cell 170, information aboutavailable space on primary volume 190 and receive, from storageoperation cell 175, information that client 185 data has been migratedto secondary storage 210. In some embodiments, data may be received frommultiple storage operations cells.

One problem traditionally associated with data integrations, such as SRMand HSM data integration, is host identification. For example, variousclients or hosts throughout an enterprise often have the same name ornetwork⋅ identifier. A client at a site in Maine may have the same nameor network identifier as a client in New York. With respect tointegrating data from these two clients, prior art systems were thusunable to reliably distinguish between the client in Maine and theclient in New York. Similarly, SRM data related to a particular clientis not correlated to HSM data related to the same client in prior artsystems. HSM systems are distinct from SRM systems and prior art systemswere not able to determine whether, for example, a Host A identified bySRM information in a network, was the same Host A identified by HSMinformation in the network. A similar problem exists, for example, whenSRM systems identify a client as Host A and HSM systems identify aclient as Host B, however, Host A and Host B are actually the samephysical machine or device. The present system overcomes theselimitations and addresses this problem by, among other methods,normalizing data received, step 230.

Clients, hosts, network devices, or other types of network elements maybe assigned a unique network identifier or other identifier orcharacteristic that identifies and distinguishes between separateentities. In some embodiments, the network identifier may be generatedwhen a component is associated with the network, for example viaauto-detection and assignment routines performed by a storage manager orother component. In other embodiments, identifiers are assignedmanually. In yet other embodiments, identifiers may be generatedaccording to characteristics of the network device or element, forexample based on a processor serial number, IP or other network address,memory configuration, hardware configuration, or other similardistinguishing characteristics and information. Such identifiers mayalso be derived by employing hashing routines or by being based on otherindicia such as hardware configuration or time of creation. In someembodiments, such identifiers may include Globally Unique Identifiers(GUIDs) generated by an operating system or other application program.

Identifiers associated with network elements may be tracked or monitoredby storage managers in storage manager databases. Thus, for example,when HSM modules or other logic in storage operation cell 175 obtaininformation about client 186, this information may be associated withthe identifier for client 186 in addition to whatever name client 186may have within the HSM system. For example, client 186 may be known bythe HSM modules as Host B. Similarly, when SRM modules or other logic instorage operation cell 170 obtain information about client 185 instorage operation cell 170, this information may also be associated withthe identifier for client 185 in addition to whatever name client 185may have in the SRM system. For example, client 185 may be known by theSRM modules as Host A.

When the SRM data and the HSM data is propagated to and received bymaster storage manager 215, the SRM data and HSM data pertaining toparticular network elements may be transmitted with the associatednetwork identifiers of those network elements. Thus, for example,storage manager 215 may normalize data related to Host A and Host B byusing the network identifiers and other metadata received with the SRMand HSM information to determine if Host A and Host B are actually boththe same physical machine, client 185. Similarly, the primary copyassociated with Host A and the primary copy associated with Host B maybe identified as the same (or different) primary copy on the samephysical media, primary volume 190 using the network identifier (e.g., aGUID).

At this point, storage characteristics, attributes, metrics, and otherfactors associated with the data received may be identified (step 235).For example, master storage manager 215 may compare SRM or HSM dataagainst certain metrics stored in master storage manager database 220.SRM data related to the total amount of disk space remaining in primaryvolume 190 thus may be evaluated against a threshold or other criteria.For example, a service level agreement (“SLA”) may require that primaryvolume 190 satisfy a threshold of having 20% available free space toguard against failure due to lack of storage capacity. HSM data relatedto secondary storage may also be evaluated against a threshold or othercriteria. For example, an SLA or an administrator preference may requirethat data older than a given time period be migrated from secondarystorage volume 211 to other storage or from primary volume 191 tosecondary storage volume 211. Alternatively, storage characteristicsrelated to a particular volume or network element across the enterpriseor a plurality of storage operation cells may be evaluated.

Thus, at this point, the system may determine whether the data receivedsatisfies a storage-related criteria or other criteria (step 240), andperforms an action or trigger event in response to the criteria beingsatisfied (step 245). For example, if a storage policy, rule, or otherdefinition indicates that an SLA or other preference specifying aparticular criteria, such as a storage threshold for a particular volumehas been met, then the system may perform a corresponding action, suchas migrating data from the volume to other storage to reduce usage onthe volume to comply with the specified SLA range. As another example,the system could trigger an event when data is not adequately backed upacross the enterprise, for example when only a single copy of dataexists or is not adequately saved as other copies on other volumes asspecified by a storage criteria. Such events may be based in whole or inpart on any suitable SRM thresholds associated with primary data or HSMthresholds associated with secondary data (or vice versa).

In some embodiments, the system uses data from one or more storageoperation cells may be used to advise users of risks or indicatesactions that can be used to mitigate or otherwise minimize these risks,and in some embodiments, dynamically takes action to mitigate orminimize these risks. For example, a master storage manager 215 mayinclude a variety of policies or definitions specifying storage-relatedrisks, risk criteria, or other use tolerances with respect to storage ofelectronic data. For example, a policy may indicate that the primaryvolume should always maintain 20% free space, that snapshots of data onthe primary copy should be taken every hour and stored on a secondaryvolume for at least 1 week, that backup data from a secondary volumeshould be aged and migrated to other storage after one month andretained for one year, etc. A policy might also indicate informationrelated to an SLA or user preference.

For example, a policy may indicate that data on a secondary volumeshould always have a certain level of availability and be able to berestored within a given time period, that data on a secondary volume maybe mirrored or otherwise migrated to a specified number of othervolumes, etc. This may be determined for example, by autodiscoverytechniques that make the system aware of the storage policies associatedwith each particular type of storage device within the network. Thus,when data is correlated from SRM, HSM, and other storage operation cellsto identify and normalize relationships between hosts, storage volumes,and other network elements as previously described herein, the systemidentifies characteristics or other aspects of the data received thatsatisfy these types risk criteria. Thus the system might determine thatprimary copy A has only 15% free space left on its current storagevolume and that a secondary storage volume storing snapshot data forprimary copy A is unavailable and thus SLA criteria with respect toagain primary copy A data is not being met.

In some embodiments, the system notifies the user of these conditionsand may suggest (or automatically implement) solutions to mitigate orotherwise address or minimize the risks to the data. Thus, for example,the system would indicate that data from primary copy A should bemigrated to secondary storage in order to free space on the storagevolume and that snapshots of data should be routed or directed to analternate or failover secondary storage volume while the other storagevolume(s) are unavailable. In some embodiments, a storage managercomponent may also take actions or other steps to mitigate or otherwiseaddress or minimize the risks to the data evidenced by the condition(s).Index data specified a in a risk criteria or other definition orcriteria indicating actions to take in the event of certain risks orother conditions being indicated by data collected from the storageoperation cell(s). In some embodiments, the system analyzes applicationconfiguration data to determine whether a risk is present. Thus, a riskmight be present and a risk criteria satisfied if an Exchange serverconfiguration or setup does not satisfy a model or exemplaryconfiguration according to a storage policy or definition.

FIG. 5 presents a flow diagram of a method to calculate storage metricsassociated with storage operation cells according to an embodiment ofthe invention. In some embodiments, data from storage operation cells iscorrelated against various storage metrics to generate an indication ofa state of data storage in the system. Thus, for example, informationabout the number or age of backup copies, the availability of storeddata, and other information related to data protection can be weightedto generate a storage metric for a level of data protection for primarycopies, secondary copies, in network elements, etc.

The system receives data from storage operation cells, step 250, and maynormalizes the data received, step 255. For example, a master storagemanager or other storage manager receives information related to storageof electronic data in storage operation cells of the system. Informationis received regarding data protection over time, space on storagevolumes, data redundancy (such as a number of multiple copies beingstored), failures and successes in storage operations, types of storageoperations, recoverability time, migration level, compliance level,performance of data path, etc.

The system may generate and store an indication of a state of storage inthe system, step 260. For example, information received may be assignedvarious weights or values according to criteria specified in the storagemanager index. This information may then be manipulated or otherwiseanalyzed to generate a numerical indication of a storage related metricof the system. A primary volume with 50% free space may be allocated 2points, error-free storage operations for the primary volume over a 12hour period may be allocated 3 points, hourly snapshots of the primaryvolume data stored to secondary storage may be allocated 4 points, anavailability time of 10 minutes may be allocated 2 points, and otherstorage-related criteria for a volume or data may be allocatedadditional metrics or values. Thus, a metric or other indication for aparticular volume can be generated. Users can quickly determine levelsof data protection or other storage-related criteria being quantifiedand tracked between volumes and other network elements. A user candetermine whether data stored on one primary copy is better protectedthan data stored on another primary copy by comparing the storagemetrics for the two volumes.

In some embodiments, the system may also determine whether a metric orother indication satisfies a particular storage criteria, step 265, andperforms an action as a result, step 270. For example, as previouslydescribed, a storage policy or other definition might indicate that astorage manager should initiate an HSM-related storage operation if astorage metric or other indication drops below or otherwise fails tosatisfy a specified criteria such as a threshold of data protection. Insome embodiments, the values and other metrics, weights, or criteriagiven to information received from storage operation cells is related tometrics or other criteria specified in an SLA.

In some embodiments, risk factors or other data protection operationsmay be quantified into certain measurable service or risk levels forease of comprehension. For example, certain applications and associateddata may be considered to be more important by an enterprise than otherdata and services. Financial compliance data, for example, may be ofgreater importance than marketing materials, etc. Network administratorsmay assign priorities or “weights” to certain data or applications,corresponding to its importance (priority value). The level ofcompliance with the storage operations specified for these applicationsmay also be assigned a certain value. Thus, the health, impact andoverall importance of a service on an enterprise may be determined, forexample, by measuring the compliance value and calculating the productof the priority value and the compliance value to determine the “servicelevel” (sometimes referred to as a “Q-factor” in the CommVault QiNetixproduct) and comparing it to certain operational thresholds to determineif the operation is being performed within a specified data protectionservice level.

As mentioned above, the service level value may be used as a metric todetermine how well data protection operations are functioning within acertain portion of a network such as a storage operation cell 50. It mayalso be viewed as point in time a measurement of data protection servicelevel. Generally speaking, the data protection service level may bedefined in terms of a range of varying time periods including short-termdata protection operations (e.g., days, hours, weeks), near term dataprotection operations (e.g., weeks, months) and long-term dataprotection operations (e.g., months, years) or any combination of thesedepending on the service to be analyzed. For example, a data protectionmetric concerned only with short-term storage performance may be definedbased mainly on the performance of short-term data operations.Conversely, a data protection metric concerned only with long-termstorage performance may be defined based mainly on the performance oflong-term data operations. A data protection metric concerned withoverall or “composite” performance may be based on the performance oflong-term, near term and short-term data operations. The emphasis ordegree to which certain operations contribute to the value of metric maybe chosen by a network to meet specific needs. For example, theperformance of short-term data operations may contribute 70% of the datametric value whereas near term operations may contribute 20% andlong-term operations may contribute 10% or vice versa depending on thedesired emphasis in the metric. Such weights or contribution levels maybe set by a user or network administrator.

Moreover, some embodiments may include preconfigured profile types basedon the type of entity the system in managing. Thus, an administrator mayindicate the type of business or entity using the storage system and thevarious storage metrics and parameters used to evaluate and otherwiseindicate information regarding the health and other characteristics ofthe system may be automatically configured. In some embodiments,administrators may use these preconfigured settings as a baseline andfurther customize weightings and other criteria used to calculatestorage metrics according to user preferences, business needs, SLAs,etc. For example, a network administrator may choose from a list of suchprofiles that may include for example, financial, medical, legal, etc.,each having metric weights for various storage parameters preconfiguredfor that business type.

This feature is desirable in part because certain businesses havedifferent storage needs, and the weights given to certain storageparameters, and thus the way in which one or more storage metrics arecalculated may necessarily be somewhat different depending on businesstype. Assume, for example, the following storage parameters are used incalculating a storage metric: 1) data availability, 2) retention period,3) protection level (number of copies), 4) disaster recovery level,hardware load, and 6) network load. A financial services company, a lawfirm, a hospital and a small retail business all may have somewhatdifferent storage needs as represented according to these metrics. Thefinancial services company may be very concerned with availability,protection level and disaster recovery level, whereas a hospital may bemore concerned with retention period and hardware load. Thus,appropriate storage metrics for the hospital may emphasize retentionperiod in its calculations (e.g., by heavily weighting this storagecharacteristic), whereas a financial services company may only assignretention period a mid-level weighting factor, etc. Similarly, a smallbusiness may care little about hardware and network load, but beconcerned with availability, and thus weight these parametersaccordingly. The law firm may want different weights on all parameters.Thus, by choosing parameter weights appropriately, the displayed storagemetric may accurately reflect the overall health and performance of thestorage system relative to business priorities, thus providing the userwith a helpful indicia of the storage network operation and dataprotection level.

In some embodiments, the storage management system may includepreconfigured business or entity profiles that are tailored to certainbusiness or entity types that allow a user to choose a profile based onbusiness or entity type without having any specific knowledge of storageor business needs. These preconfigured profiles may contain metricweights appropriate for the specified business or entity type. From theexample above, a user may choose a financial services profile thatspecifies the appropriate weights for that business type. This relievesthe user of the burden of becoming familiar with the specifics of thestorage system or the businesses storage needs to configure the system.Moreover, in some embodiments, the system may include auto-discoveryroutines automatically seek out and evaluate the user's network and makeprofile recommendations based the particulars of that discovered network(either automatically or with some user input).

Generally speaking, in some embodiments, a short-term data protectionservice level may be computed by considering various short-term dataprotection aspects or considerations. For example, a priority may beassigned to each aspect on a predetermined scale (e.g., from 1 to 9,where 9 representing a service of least importance). After a priority isassigned, a weight for the aspect may be calculated as (10—AssignedPriority). In one embodiment, percentage points for each aspect may becalculated as weighted average among various aspects that are underconsideration according to the following formula:

Sdp=Sslp*(Wa/ΣWap)  (2)

where Sdp is the short term data protection service level, Sslp is theis the short term data protection service level percentage, Wa is theweight for a certain aspect under consideration and Wap is the weightfor all relevant short term data protection aspects (which may besummed).

Some of the various aspects that may be considered when computing theshort-term data protection service level may include:

-   -   1. The threshold for not having any data protection operation in        a given number of days. If no data protection operation has        occurred in the past specified number of days, percentage points        for this aspect may be subtracted from the total percentage        points or otherwise taken into account when calculating        short-term data protection service level.    -   2. The threshold for not having full data protection operation        (e.g., a complete backup operation) in a given number of days.        If there is no full data protection operation in the past        specified number of days, percentage points for this aspect may        subtracted from the total percentage points or otherwise taken        into account when calculating short-term data protection service        level.    -   3. The data protection cycle redundancy. If there is no        redundancy in the data protection pathways, percentage points        for this aspect may be subtracted from the total percentage        points or otherwise taken into account when calculating        short-term data protection service level.    -   4. Threshold for not encountering any failures in data        protection operations in a given number of days. If one or more        data protection operation failures is encountered in a specified        period, percentage points for this aspect may be subtracted from        the total percentage points or otherwise taken into account when        calculating short-term data protection service level.

Similarly, long-term or near term data protection service level may becomputed using the formula set forth above by taking variouslong-term/near term data protection aspects in to account which mayinclude the following:

-   -   1. Weekly data protection for a given number of weeks (e.g., for        calculating near term service level). If a data protection        operation is missing in a week within the specified number of        weeks, then the percentage points for this aspect may be        subtracted from the total percentage points or otherwise taken        into account when calculating long-term data protection service        level.    -   2. Monthly data protection for a given number of months (e.g.,        for calculating near term or long term service level). If a data        protection operation is missing in a month within the specified        number of years, then the percentage points for this aspect may        be subtracted from the total percentage points or otherwise        taken into account when calculating long-term data protection        service level.    -   3. Yearly data protection for a given number of years (e.g., for        calculating near term or long term service level). If a data        protection operation is missing in a year within the specified        number of year, then the percentage points for this aspect are        subtracted from the total percentage points or otherwise taken        into account when calculating long-term data protection service        level.

If an overall performance metric is desired, it may be computedaccording to the following equation:

Sop=(SF1)(Sdp)+(SF2)(Sntp)+(SF3)(Sltp)  (3)

where Sop is the overall performance, Sdp is the short term performancemetric; SF1 is a scaling factor assigned to provide a certain weight toSdp; Sntp is the near term performance and SF2 is the scaling factorassigned to provide a certain weight to Sntp; Sltp is the long termperformance metric and SF2 is a scaling factor assigned to provide acertain weight to Sitp.

Furthermore, it will be understood from the foregoing that the suchperformance metrics may be calculated on a scalable basis such that manyshort-term, near term, or long-term or composite metrics representativeof the performance or various portions of a network may be combined andaggregated as desired to obtain a service level performance values thatrepresent the operational status of a large, small or intermediate sizenetwork.

Moreover, additional metrics different from the illustrative examplesset forth above also be used if desired in the calculation of servicelevel values. Some examples include: 1) disaster recovery, 2) storageutilization, 3) data migration level, 4) availability level, 5)compliance level, and 6) the number of available data paths, etc. mayalso be included in the calculation of a service level.

If desired, these metrics may be arranged individually and presented ingraphical form to the illustrate conformance level of each of theseindicators rather than (or in addition to) being included in a certainmetric as described above. For example, FIG. 6 presents a bar chart 300that illustrates the calculated service level associated with eachcorresponding metric. For example, moving from left to right thefollowing metrics are illustrated, application server level 301, datapath level 302, compliance level 303, availability level 304, migrationlevel 305, storage use 306, and overall protection level 307. Thedesired service level for each metric is illustrated by the dotted lines308. If the bar graph for a particular metric is below dotted line 308,the service is underperforming, if at or above dotted line 308, theservice meets or exceeds expectations. In some embodiments, users mayestablish the desired service level for each metric through the userinterface of FIG. 7 (described in more detail below).

Some of the factors considered in calculating service 301 may includedata transfer rate throughput and congestions levels. Data path level302 may be calculated taking into consideration the number of availabledifferent data paths, SAN configuration and reporting, and backupnetwork performance. Compliance level 303 may be calculated taking intoconsideration data path encryption, encryption security, searchabilityof data and the degree to which the data is indexed. Availability level304 may be calculated taking into consideration where data is stored andhow fast it may be retrieved. Migration level 305 may be calculated bytaking into consideration the state of secondary copies, the age of thesecondary copies and the level of migration. Storage use level 306 maybe calculated by taking into consideration HSM and SRM storage use, andoverall protection level 307 may be calculated by taking into accountall or some of the foregoing factors based on system or enterpriseneeds.

In accordance with one aspect of the present invention, networkadministrators or other users may specify parameters associated withdata protections through a graphical user interface such as the onedepicted in FIG. 7. As shown, Interface 400 may include varioususer-defined or user-selectable fields relating to permissible values ofranges for data protection parameters for short and long term storage.Such an interface may include, for example, fields for entering theacceptable overall service level (field 401, e.g., a storage metric),the weights assigned to the category of coverage (short term or longterm, field 402), certain short term storage parameters which specifyoperations such as the time period in which at least one data protectionoperation is performed (field 403), time period in which the last fulldata operation is performed (field 404), time period in which more thanone full data protection operation is performed (field 405), and thetime period in which there has been no data protection failure (field406). Each of the fields may also be assigned a priority level throughinterface 400 (generally labeled field 407). The percentage each factorcontributes to the storage metric is displayed in field 408. Moreover,as shown, interface 400 may also include fields for activating ordeactivating such fields to control whether such parameters are actuallyused in calculating a particular service level.

FIG. 8 presents a flow diagram generally illustrating a method tocalculate data costing and data availability associated with storageoperation cells according to an embodiment of the invention. Datareceived from storage operation cell 50 may used in conjunction withhardware-related information and other information about networkelements to generate indications of costs associated with storage ofparticular data in the system or the availability of particular data inthe system.

At step 575, certain network elements such as storage volumes, storagedevices, computers, switches, routers, bridges, SANs, cabling, and othertypes of network infrastructure, may be identified. In some embodiments,information regarding network elements is manually obtained, for exampleas input by a system administrator or from a supplied configurationfile. In other embodiments, information regarding network elements maybe dynamically obtained or generated. For example, a storage operationcell may contain network element detection software or other modulesthat dynamically perform network discovery operations to locate andidentify network elements. In various embodiments, network elementdiscovery may occur according to a predefined schedule, at startup, orwhenever a new network element is incorporated or otherwise associatedwith a storage operation cell in the system. In some embodiments,storage operations cells throughout the storage operation cell hierarchymay propagate network element discovery information to a master storagemanager for update.

At step 580, characteristics or metrics associated with the networkelements may be identified. A storage manager index may containcharacteristics or metrics associated with information identifyingpotential network elements. Thus, the system may receive informationidentifying a particular network element and then associate informationsuch as a characteristic or metric with that network element for furtheruse generating an indication of storage cost or data availability. Forexample, a tape storage device might be associated with a storage costof 3 cents/MB and a data availability level of 40 MB/sec. A section offiber optic cabling could be associated with a throughput speed of 3Gbits at a particular cost, etc.

As previously described herein, the system may also receive other datafrom storage operation cells (step 585), and may normalize the data(step 590). For example, a master storage manager may receive andnormalize SRM and HSM information. Exemplary information may includevolume information, user information, business logic information such asdepartment or project identifier or association, etc. Information froman SRM storage operation cell may be received indicating that a primarycopy A (or subsets of data associated with primary copy A such as afiles, folders, mailboxes, etc.) belongs to the marketing department andis used by three users A, B, and C. User A may generate 70 percent ofthe data on primary copy A. Primary copy A may be stored on primaryvolume A and may relate to data from host A. Host A may correspond toHSM storage operation cell host B. Primary volume A may correspond toprimary volume B and which is backed up to secondary storage volume C.

The additional data received from the storage operation cells may becorrelated at step 585 with the network element information from steps575 and 585 (step 595). For example, it may be determined that a harddrive from step 575 corresponds to the storage volume for primary copy Aused by users A, B, and C of the marketing department. Primary copy Adata from the hard drive may be backed up to a tape drive alsodiscovered in step 575. The system also correlates that the snapshotinformation or other backup information received from an HSM storageoperation cell regarding a secondary storage volume corresponds to thetape drive.

Thus, the system generates and stores data characteristics and metricsassociated with the network elements of step 575 and the additionalinformation received from the network cells in step 575 (step 600). Forexample, knowing that primary copy A of the marketing department isstored on a hard drive allows the system to generate a storage cost forthe marketing department primary copy data based on informationgenerated in step 580. A cost per megabyte for primary storage of themarketing department may be calculated and allocated among various usersaccording to additional information received about use patterns byparticular users. A particular user can be monitored, blocked, orrestricted etc. according to data usage thresholds or other criteria.Additionally, the secondary storage costs of marketing department datacan also be generated since the primary volume data may be correlated tosecondary storage elements using the SRM and HSM data.

Exemplary information generated could include how fast a particulardepartment is using up available storage space, how long data would taketo recover over a particular network pathway from a particular secondarystorage device, costs over time, etc. The system may, for example,calculate based on network element and pathway information and metricsacross the enterprise, whether a particular SLA is being met withrespect to recovery times for marketing department data, data of aparticular user, data for a particular project, etc. In someembodiments, media usage may be predicted according to data use,availability, or cost. For example, a pattern of use by the marketingdepartment may predict or otherwise indicate that the primary copy Awill need to be migrated to a larger primary storage volume in the next6-8 months at a particular cost per MS.

Moreover, in some embodiments, such information may be used to determineor predict the overall cost associated with the storage of certaininformation. The cost associated with hosting a certain application maybe based, at least in part, on the type of media on which the dataresides. For example, an Exchange Server may be provisioned with 200 GBas primary storage and 2 TB of secondary storage. The storage of data oneach type of storage media may be assigned a certain cost parameter suchthan overall cost may be calculated. For example, the cost of storingdata on primary storage may be 20 cents per Megabyte, primary storagesuch as LTO may have a 15 cents per Megabyte storage cost and secondarystorage such as DLT may have a 10 cents per Megabyte Storage cost. Thus,overall cost associated with the Exchange Server application may becalculated by multiplying the amount of data stored on each media typeby its associated cost and adding the results to obtain the sum. Thisallows network administrators to quickly and accurately determine thecurrent cost of particular application as well as to predict the cost ofcontinuing to host the application as it grows (or becomes smaller).

This method may also be used to determine storage cost of over a periodof time. For example, the amount of data stored on a certain media maychange pursuant to, among other things, a storage policy andfluctuations in application activity. Cost over a specified time periodmay be calculated such that it takes into account the movement of datafrom a more expensive storage medium to a less expensive one as well asany increase in cost due to any change in the amount of storage requiredto service an application. This may be done, for example by multiplyingcost by capacity over a specified period of time.

The costing feature of the present invention may include an interfaceconsole or other graphical display that allows users to display anddetermine costs associated with a particular application. Such a consolemay allow a user to select (and possibly combine) certain applicationsto determine the storage cost associated with that application. Forexample, the console may include a desktop with icons or other indictorrepresenting certain applications. A user may, for example, click on oneof these applications, and determine data storage costs (or otherinformation by selecting various options from a pull down menus). Theinterface may also allow the user to combine certain applicationtogether to obtain an overall performance cost to drill down toparticulars within that application to determine costs associated with aparticular aspect of that application (not shown).

Moreover, the costing interface of the present invention may includefields that permit associations for creating a billable entity forallocating costs based on data use. For example, if three differentgroups such marketing, sales and accounting, access the same data, thecost for maintaining that data may be divided among them depending, oncertain criteria such as frequency of use. Thus, for example, if salesuses the application 80% of the time while marketing an accounting onlyuse the application 10% of the time, the cost of maintaining theapplication may allocated accordingly. Such allocations may be performedmanually, or automatically (for example, at step 595).

Moreover, storage devices may be assigned to a particular cost categorywhich is indicative of the cost of storing information on that device.For example, a network may employ many different types of storagedevices each of which may have a different associated cost. By assigningeach device to a different cost category, the system may determine,based on device identity, the cost associated with storing data on thatdevice. Thus, costing categories allow user to quickly determine theoverall cost associated with certain data as it evolves through itslifecycle. It also allows user to accurately predict the cost associatedwith a proposed or anticipated storage operations based on the storageresources to be used. Moreover, certain billable entities may be chargedappropriately based the capacity of a storage device used by thatentity.

FIG. 9 shows a console 900 suitable for allowing system users to createcost categories. As shown, cost console 9 may include a name field 901,a cost per unit of storage field 902, a description field 903 andavailable device field 904. In operation a, user may select a storagedevice from field 904 (which may be automatically populated with networkdevices using auto-discovery techniques) and assign a cost to thatdevice. Alternatively, default costs may be assigned to the detectednetwork devices from a database containing such information. Moreover,if desired, a user may customize the characteristics of devices byaltering the default values, or by entering new network devices in namefield 901 and specifying cost values.

Referring to FIG. 10, some of the steps involved in automaticallyassigning cost categories may be as follows. Autodetect the networktopology including the various storage devices and storage media for agiven network, which may be a subset of a larger network (step 1010). Insome embodiments, this may involve the creation of network topologymodel. Next, the identification of each discovered storage device may becompared to a database of known elements to determine if there arematches (step 1015). The information in the database may include costinformation, failure rate, throughput and other performance metrics.Assuming matches are found, the information in the database iscorrelated with the model of the network topology to createparticularized or parameterized instances of the discovered devices inthe topology or model (step 1020). At step 1025, a user may manuallyinput parameter information into the topology for storage devices notfound in the database. Once the model/topology is created the system maymonitor the devices to determine if the device is underperforming orotherwise deviating from expected baselines or thresholds (step 1030).For example, the system may monitor device throughput according tovalues associated with each device as populated in the topology modeland trigger alerts or take other corrective actions if throughput fallsbelow a given level or other value for a particular device or data path.Similarly, during the auto-discovery phase, an expected failure rate forstorage operations for a particular device may be specified and thesystem will trigger alerts or take other corrective action (e.g.,failing over storage operations to an alternate device) in the eventthat failure rates for the particular device exceed the specifiedexpected failure rate. At step 1035, the system may continue to monitorthe network to track the costs associated with data storage and use asfurther described herein.

Systems and modules described herein may comprise software, firmware,hardware, or any combination(s) of software, firmware, or hardwaresuitable for the purposes described herein. Software and other modulesmay reside on servers, workstations, personal computers, computerizedtablets, PDAs, and other devices suitable for the purposes describedherein. Software and other modules may be accessible via local memory,via a network, via a browser or other application in an ASP context, orvia other means suitable for the purposes described herein. Datastructures described herein may comprise computer files, variables,programming arrays, programming structures, or any electronicinformation storage schemes or methods, or any combinations thereof,suitable for the purposes described herein. User interface elementsdescribed herein may comprise elements from graphical user interfaces,command line interfaces, and other interfaces suitable for the purposesdescribed herein. Screenshots presented and described herein can bedisplayed differently as known in the art to input, access, change,manipulate, modify, alter, and work with information.

While the invention has been described and illustrated in connectionwith preferred embodiments, many variations and modifications as will beevident to those skilled in this art may be made without departing fromthe spirit and scope of the invention, and the invention is thus not tobe limited to the precise details of methodology or construction setforth above as such variations and modification are intended to beincluded within the scope of the invention.

What is claimed is:
 1. A method of providing a unified view of primaryand secondary storage resources comprising: monitoring primary dataoperations generated by one or more applications executing on one ormore client computing devices in a primary storage system, the primarydata operations storing primary data in one or more primary storagedevices in at least a first format that is a native format associatedwith the applications that generated the primary data; performing, withthe one or more media agents comprising computer hardware, secondarydata operations that copy the primary data to create a secondary copy onone or more secondary storage devices, the secondary copy stored in asecondary format that is different than the native format; generatingone or more primary weighting factors associated with the usage of theone or more primary storage devices; generating one or more secondaryweighting factors associated with the usage of the one or more primarystorage devices; determining a data protection service level based atleast in part on the one or more primary weighting factors and based atleast in part on the one or more secondary weighting factors; anddisplaying in a user interface a unified view of the primary dataoperations and the secondary data operations, the user interfaceconfigured to display at least the data protection service level.
 2. Themethod of claim 1 wherein the data protection service level comprises ashort-term data protection service level and a long-term data protectionservice level.
 3. The method of claim 2 wherein the user interfacereceives input from a user that adjusts one or more storage parametersassociated with the short-term data protection service level and thelong-term data protection service level.
 4. The method of claim 1wherein the one or more primary weighting factors comprise theavailability of the one or more primary storage devices.
 5. The methodof claim 1 wherein the wherein the one or more secondary weightingfactors comprise the age of the secondary copy on the one or moresecondary storage devices.
 6. The method of claim 1 wherein the one ormore secondary weighting factors comprise the number of secondary copieson the one or more secondary storage devices.
 7. The method of claim 1wherein a first media agent is associated with a first storage managerof a first data storage cell and a second media agent is associated witha second storage manager of a second data storage cell.
 8. The method ofclaim 1 wherein the user interface data displays SRM operations and HSMoperations.
 9. The method of claim 1 further comprising storing theplurality of secondary weighting factors in a storage policy, thestorage policy comprising a data structure that further comprises a setof preferences for performing one or more secondary storage operations.10. The method of claim 1 further comprising executing an auto-discoveryroutine configured to automatically evaluating one or more storagepolicies to determine the one or more primary and secondary weightingfactors.
 11. A system of providing a unified view of primary andsecondary storage resources comprising: at least one storage managerthat monitors primary data operations generated by one or moreapplications executing on one or more client computing devices in aprimary storage system, the primary data operations storing primary datain one or more primary storage devices in at least a first format thatis a native format associated with the applications that generated theprimary data; one or more media agents comprising computer hardware thatperform secondary data operations that copy the primary data to create asecondary copy on one or more secondary storage devices, the secondarycopy stored in a secondary format that is different than the nativeformat; the storage manager generates one or more primary weightingfactors associated with the usage of the one or more primary storagedevices and one or more secondary weighting factors associated with theusage of the one or more primary storage devices; the storage managerdetermines a data protection service level based at least in part on theone or more primary weighting factors and based at least in part on theone or more secondary weighting factors; and the storage managergenerates a user interface with a unified view of the primary dataoperations and the secondary data operations, the user interfaceconfigured to display at least the data protection service level. 12.The system of claim 11 wherein the data protection service levelcomprises a short-term data protection service level and a long-termdata protection service level.
 13. The system of claim 12 wherein theuser interface receives input from a user that adjusts one or morestorage parameters associated with the short-term data protectionservice level and the long-term data protection service level.
 14. Thesystem of claim 11 wherein the one or more primary weighting factorscomprise the availability of the one or more primary storage devices.15. The system of claim 11 wherein the wherein the one or more secondaryweighting factors comprise the age of the secondary copy on the one ormore secondary storage devices.
 16. The system of claim 11 wherein theone or more secondary weighting factors comprise the number of secondarycopies on the one or more secondary storage devices.
 17. The system ofclaim 11 wherein a first media agent is associated with a first storagemanager of a first data storage cell and a second media agent isassociated with a second storage manager of a second data storage cell.18. The system of claim 11 wherein the user interface data displays SRMoperations and HSM operations.
 19. The system of claim 11 furthercomprising storing the plurality of secondary weighting factors in astorage policy, the storage policy comprising a data structure thatfurther comprises a set of preferences for performing one or moresecondary storage operations.
 20. The system of claim 11 furthercomprising automatically evaluating one or more storage policies todetermine the one or more primary and secondary weighting factors.